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Abstract 

This paper discusses one of the fundamental issues in education and social program evaluation: the proper role 
for an evaluator. Based on respective and comparative analysis of five theorists’ positions on this fundamental 
issue, this paper reveals how different perspectives on other fundamental issues in evaluation such as value, 
methods, use and purposes can result in different roles for evaluators, and how such difference can affect 
evaluators’ responsibilities in different stages of an evaluation. Then the paper proposes its own resolution of the 
issue of evaluator’s role and discusses its implication and limitations. 
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1. Introduction 

Fundamental issues in evaluation include the purpose of evaluation, the nature of evaluation, the best methods, 
strategies and tools of conducting evaluation, the practical concerns such as politics, clients and resources and 
their influence on evaluation, as well as the roles, ethics and responsibilities of evaluators. Fundamental issues 
are defined as “those underlying concerns, problems, or choices that continually resurface in different guises 
throughout our evaluation work.” (Smith, 2008, p.2) Although those fundamental issues will resurface 
periodically in the field of evaluation in new forms and cannot really be solved once and for all; the awareness of 
the recurring nature of such fundamental issues can help one view the current problems in evaluation from a 
better historical perspective. By identifying and examining such fundamental issues, evaluators can have a 
deeper understanding of their importance, constraints and alternative solutions, thus propose a more effective, 
yet still impermanent resolution for the existing problems. 

1.1 Evaluator’s Role as a Fundamental Issue 

The fundamental issue discussed in this paper is the role for an evaluator. Over the years, many evaluation 
theorists have proposed different roles for evaluators. For example, Scriven sees evaluator as a “judge” who 
justifies the value of an evaluand and offers his summative judgment in the final report; while Stake believes an 
evaluator should be a “program facilitator” who works with different stakeholders and assists them to “discover 
ideas, answers, and solutions within their own mind”. Campbell prefers a “methodologist” role for an evaluator, 
advocating rigorous experiment design that yields strong causal inferences; but Wholey believes an evaluator 
should be an “educator”, whose role is to infuse useful information to the potential users of the evaluation. The 
emphasis resides not only in the immediate outcome of a program, but also in the inputs, implementation and 
long-term outcome of the program. 

However, terms such as “judge”, “methodologist” and “educator” are just metaphors to facilitate understanding 
and cannot always accurately describe the role an evaluator plays during an evaluation. In fact, evaluators often 
play different roles in different phases of an evaluation. For example, an evaluator can be a judge during the 
phase of selecting criteria of merit, a methodologist when collecting data, a program facilitator during the 
program implementation, and an educator during the results dissemination. The roles an evaluator takes during 
an evaluation reflect his or her beliefs in other fundamental issues such as theories, values, methods, practice and 
use, etc. Other issues such as education background, previous working experiences, the nature and setting of 
social programs might also contribute to shaping the proper roles for an evaluator. 

1.2 Importance of Evaluator’s Role as a Fundamental Issue 

Just like any other fundamental issue in evaluation, there is no final resolution to defining the proper role for an 
evaluator. However, studying the different roles an evaluator can play, as proposed by different theorists, is still 
quite important for evaluators, clients and evaluation as a profession. 

Turning first to evaluators, studying the different roles for an evaluator is actually studying the different 
approaches of conducting evaluation. An evaluator’s role is not self-claimed; rather it is defined by the things an 
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evaluator does during the evaluation. For instance, we wouldn’t use the metaphor “judge” to describe the 
evaluator’s role in Scriven’s theory if evaluators’ job doesn’t include determining the criteria of merits, setting 
comparative standards and giving a final summative judgment. Or we wouldn’t compare the role of evaluator 
in Weiss’s theory to an educator if providing “enlightenment” (Note 1) is not a primary task for her evaluators. 
The familiarity with different roles an evaluator can play allows one to take a more flexible approach to conduct 
evaluation according to specific context, the nature of social program, available resources, and different client 
expectations. 

As for the clients of an evaluation, the awareness of the different roles of an evaluator can play will help them 
select the right candidate according to their specific needs; and reach an agreement with the selected evaluator 
about his/her job responsibilities as well as the obligations clients shall make in order to facilitate the evaluation 
process. For example, for a program that does not welcome intrusion, an evaluator who prefers doing an 
experiment might not be the best candidate. For an evaluator who prefers the role as a “program facilitator”, 
program administrators should anticipate frequent meetings with the evaluator and make incremental changes 
according to his/her feedback. The basic knowledge about evaluator’s roles is especially important in today’s 
world of globalization, where evaluation in a different nation or culture becomes more common. Clarifying the 
role an evaluator should play beforehand is a good way to avoid surprise, misunderstanding, and conflict later 
on. 

Finally, evaluation as a profession will benefit from a deeper understanding of the roles of evaluators. How is an 
evaluator different from a social scientist? Can a methodologist be hired to do the job of an evaluator? What is 
the difference between evaluation and research? What are the competencies that are unique for an evaluator? 
Those questions are raised due to the lack of distinction between evaluation and other social science professions. 
Explicating the proper roles for a professional evaluator will be a good approach to address such distinction and 
solidify the status of evaluation as a profession. 

2. Theorists’ Positions on the Fundamental Issue of Evaluator’s Role 

Many theorists in the field of evaluation have different opinions regarding the proper roles for an evaluator. 
Their opinions on this issue reflect their overall philosophy about doing evaluation as well as their stances on 
other fundamental issues in evaluation. This section will first discuss the resolutions proposed by different 
theorists regarding the role of evaluator, analyzing the strength and weakness of each resolution. Then a 
comparative analysis will be conducted to study the positions across those theorists. 

2.1 Sc riven 

Scriven believes that an evaluator’s role is to investigate and justify the value of an evaluand. Such investigation 
and justification shall be supported with joining empirical facts and probative reasoning. “Bad is bad and good is 
good and it is the job of evaluators to decide which is which” (Scriven, 1986, p. 19). He rejects the notion that an 
evaluator’s role is simply to provide information to decision-makers and claims that “the arguments for keeping 
science value free are in general extremely bad” (Scriven, 1969, p.36). According to Scriven, an evaluator’s 
responsibilities during an evaluation include: 

• Determining criteria of merit from needs assessment. Criteria of merit of an evaluand should be its capacity 
to meet needs. Although an evaluator can use the results of needs assessment conducted by a program developer, 
sometimes he/she should do an independent needs-analysis. To avoid bias, Scriven advises evaluators to conduct 
“goal-free” evaluation and formulate questions by ignoring the program goals and looking for all possible effects 
an evaluand could have. 

• Setting comparative evaluation standards. A set of standards should be created by evaluators to assess the 
program performance. Such standards are used for comparison, either comparison with a set level of 
performance, or with alternative programs. The latter comparison is preferred by Scriven since he believes that 
an evaluator will usually make decisions about choosing among alternatives. 

• Assess program performance. An evaluator will need to answer both the evaluative and non-evaluative 
questions. Evaluative questions focus on the effects of the program and should be given top priority. The 
evaluator should acquire the skills to collect and analyze both experimental and non-experimental data. 

• Offering a final evaluative judgment. An evaluator should synthesize his findings into a final report and 
offer his/her summative judgment. 

Strength and Weakness of Scriven’s Position: Scriven differentiates evaluators from researchers or social 
scientists by emphasizing that the value judgment is an integral part of an evaluator’s role and grounds such role 
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in the logic of evaluation. His “goal-free” evaluation allows evaluators to identify possible side effects of the 
evaluand and address the concerns of underrepresented stakeholders. 

However, besides giving evaluators higher authority over different stakeholders in value judgment, Scriven fails 
to provide a solution to eliminate personal biases of evaluators. Metaevaluation proposed by Scriven is a good 
attempt but still it is highly subjective and requires years of experiences and expertise for an evaluator to make a 
non-biased judgment. For the novice evaluator, the decision of whose needs should be considered and which 
merit should take higher priority can still be very arbitrary. Besides, a complete goal-free evaluation is also 
highly unfeasible when an evaluator is hired by his/her clients and has an obligation to answer their specific 
inquiries. 

2.2 Campbell 

Campbell believes that evaluators should play a role of methodologist during the program evaluation (Shadish, 
1991, p.141). Evaluators should use scientific methodologies to design evaluative research that eliminate biases 
and establish a causal inference about a program and its hypothesized effects. This role of methodologist 
advocated by Campbell requires evaluators to employ a strong research design such as randomized experiment 
or good quasi-experiment to determine the causal effectiveness of the program. (Shadish, 1991, p. 129) An 
evaluator should also distance him/herself from the program stakeholders and work independently to find out the 
facts about the program. As for the dissemination of the evaluation findings, an evaluator should “write honest 
reports for peers even if they cannot do so for funders or the public.” (Shadish, 1991, p. 162) Last but not least, it 
is also the obligation of evaluators to play an active role in scrutinizing, replicating, and debating the evaluation 
results. 

Campbell’s emphasis on methods of measuring the program outcome makes him less concerned about assigning 
values to the program or facilitating the use of evaluation. As a result, he believes an evaluator is not responsible 
for doing the following: 

• An evaluator is not obligated to assign value to the program being evaluated. Valuing of evaluation results 
should be left to the political process, not researchers. (Shadish, 1991, p. 160). 

• An evaluator shouldn’t promote use of her evaluation results actively “since this detracts from the 
credibility of the more factlike findings.” (Shadish, 1991, p. 162). 

• It is up to the policy maker, stakeholders to decide how to interpret, disseminate and use the evaluation 
results. 

• An evaluator is not obligated to generate a different or modified program worth testing. Her job is simply 
testing the efficacy of existing programs. 

• An evaluator should avoid evaluating institutions, social organizations, or persons due to the almost 
inevitable corruption pressure. (Campbell, 1984, p.41). 

Strength and Weakness of Campbell’s Position: The methodologist role Campbell assigns to evaluators is 
echoed with the proposal for conducting “scientifically based evaluation” as advocated by the Department of 
Education. The role of a methodologist as defined by Campbell focuses on the internal validity of the causal 
inference while is less concerned about the prescribing values and utility of the evaluation findings therefore is 
quite suitable for an external evaluation regarding program outcome. Such role for an evaluator will also greatly 
enhance the scientific nature of evaluation as a profession. 

Nevertheless, the weaknesses are also quite obvious for such role of an evaluator. First of all, it is hard to 
distinguish evaluation from other social science research if one sees an evaluator merely as a research 
methodologist. Apparently, not every social scientist can do a good evaluation. Secondly, doing a rigorous 
experiment design is preferable but not always feasible. The cost and time for doing a randomized control 
experiment, as well as its intrusion into program might result in fewer and fewer evaluation being done due to 
the reluctance from program administrators. Last but not least, the methodologist role restricts evaluators to 
study only the outcome of the program while missing other key information such as how the program is 
implemented, or which element of the program works and doesn’t work. As a result, an evaluator couldn’t give 
advice about how to improve the program or adapt the program to fit other contexts. 

2.3 Stake 

Stake believes an evaluator should play a facilitator role during the evaluation. The evaluator should assist 
different stakeholders to “discover ideas, answers, and solutions within their own mind” by conducting 
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responsive evaluation. (Stake & Trumbull, 1982, p.l) According to Stake, the responsibilities of an evaluator 
include: 

• Identifying the stakeholders for whom the evaluation will be used: The evaluator should have a good sense 
of whom he is working for and their concerns. (Stake, 1975, as cited in Shadish 1991, p.273). Minority 
stakeholder groups should also be included to ensure justice and fairness. 

• Spending more time observing the program and providing accurate portrayals of the program using case 
studies. Because case studies reflect the complexities of the reality, they help readers to form their own opinions 
and judgments about the case and they can be “useful in theory building”. (Stake, 1978, as cited in Shadish 1991, 
p.289) 

• Conducting responsive evaluation which allows evaluation questions and methods to emerge from 
observing the program. In this approach, evaluators will orient evaluation directly to program activities than to 
the program goals and respond promptly to audience information requests. 

• Presenting his evaluation findings in the “natural ways in which people assimilate information and arrive at 
understandings” so that the writings can reach maximal comprehensibility. (Stake, 1980, p.83) 

Stake doesn’t believe an evaluator should make a summative value judgment since there is “no single true value” 
for all the stakeholders of a program. (Stake, 1975, as cited in Shadish 1991, p.274) As a result, evaluators 
shouldn’t blindly accept state and federal standards and impose treatments on local programs since such 
standards are not pluralistic and might not be in the best interest of local people. (Shadish, 1991, 279) Stake also 
believes the responsibility of synthesizing and interpreting case studies lies in the readers rather than evaluators 
and it is up to the readers to resolve any conflicting arguments. (Shadish 1991, p.293) 

Strength and Weakness of Stake’s Position: the facilitator role for an evaluator, as suggested by Stake, has two 
major strengths. First, it indicates the interest shift among evaluators from giving a summative judgment, 
whether it is a value judgment or an effect judgment, to generating useful information that can be used to 
improve the program. Secondly, it justifies new ways to conduct an evaluation (e.g. responsive evaluation, case 
study) and report its findings (e.g. narrative portrayal). 

However, Stake fails to take into account clients’ expectations about the proper role for an evaluator. Will clients 
accept the case study as the only approach of investigation? Will clients allow evaluators to start evaluations 
without preordinate questions? Is it appropriate for an evaluator to completely ignore the state or federal 
standards when evaluating local programs? All those doubts regarding the feasibility and validity of case studies 
or responsive evaluation will also undermine the social acceptance of the evaluator role proposed by Stake. 

2.4 Weiss 

Weiss emphasizes the evaluator’s special role in promoting the use of his/her evaluation results, especially in the 
policy-making process. She is frustrated about the fact that “evaluation results have generally not exerted 
significant influence on program decisions”, and she argues that evaluation should start out with use in mind and 
evaluators shouldn’t leave the use of evaluation to the natural processes of dissemination and application. (Weiss, 
1972, as cited in Shadish 1991, p. 182-183) Weiss claims that evaluation “should be continuing education for 
program managers, planners and policy makers”. (Weiss, 1988, p. 18) As a result, it seems that she sees the role 
of evaluator more as an educator, who conducts evaluation not for giving an explicit solution to a social problem, 
but for providing useful information to its potential users, policy-makers in particular. 

She urges evaluators to look beyond the instrumental use of evaluation results and conduct “enlightenment” 
research that “provides evidence that can be used by men and women of judgment in their efforts to research 
solutions” (Weiss, 1978, p.76) so as to maximize the utility of evaluation results. By doing evaluation this way, 
an evaluator should 

• Assess the likelihood that evaluation results might be used. (Shadish, 1991, p. 198) 

• Ask questions that can “provide an intellectual setting of concepts, propositions, orientations, and empirical 
generalizations” for policy making. (Weiss, 1978, as cited in Shadish 1991, p.202) 

• Use well designed qualitative and quantitative methods to conduct evaluation study with emphasis not only 
on the immediate outcome of a program, but also on the inputs, implementation and long-term outcome of the 
program. (Shadish, 1991, p.205) 
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• Draw policy implications from evaluation research by compiling separate summaries to multiple 
stakeholders with knowledge and information that best interest them. Make recommendations for future 
programs from the data of evaluation results. (Shadish, 1991, p.205-206) 

Strength and Weakness of Weiss’ Position: Weiss further differentiates the role of evaluator from the role of a 
researcher by addressing the complex political context that besets social programs. She warns evaluators against 
political naivety and urges them to do evaluation that can be used in policy-making, in the form of 
“enlightenment” rather than “instrumental use”. The educator role she assigns to evaluators reflects her 
pragmatic view of evaluation and suggests a new mode that evaluation can be used. 

However, the role of evaluator proposed by Weiss has some intrinsic flaws. First, such role fails to take into 
account of the variety of different contexts (ironically). For instance, the decision for the state or federal 
government to hire an educator is often not for the purpose of “being educated”, but to get concrete data 
regarding the program effect. The proposal to conduct “scientifically based evaluation” made by the Department 
of Education is a good example of that. As a result, an evaluator who uses case studies to describe the program 
input, implementation and long-term effect might not by appreciated by policy-makers in this context. Secondly, 
her emphasis on providing information to policy-makers poses the danger of evaluators becoming the servant of 
that particular stakeholder group. What should be the role of an evaluator when the interests of different 
stakeholder groups conflict with each other and speaking for the underrepresented group might limit the use of 
evaluation results in the policy-making process? 

2.5 Rossi 

Rossi didn’t give an explicit definition about the role of evaluators. Rather, the roles an evaluator shall play 
might vary according to different stages of evaluation. For example, in the program conceptualization stage, an 
evaluator sometimes takes the role of a social scientist, incorporating social science theories into the 
development of an intervention model. (Shadish, 1991, p.389-391) In the stage of program implementation, an 
evaluator works as a program administrator, making sure the program is implemented as expected so as to “rule 
out faulty implementation as a culprit in poor program outcome”. (Shadish, 1991, p.381) Besides, the operational 
data collected this way can also be useful for the future dissemination of the program. When determining the 
program utility, an evaluator will take the roles of a methodologist and a project manager, who selects and 
applies appropriate research methods to assess the impact of program intervention as well as conducts efficiency 
analysis about the program such as cost-benefit and cost-effectiveness analysis. (Rossi & Freeman, 1985, 
p.327-328) 

The different types of social programs will also affect the roles an evaluator plays during the evaluation. Rossi 
has categorized social programs into three types: innovative, established, and fine-tuning programs. For instance, 
when evaluating innovative programs, much emphasis is given to the conceptualization of the program. (Shadish, 
1991, p. 404) An evaluator’s responsibility will include setting program objectives and constructing an impact 
model between program objectives and activities, which should be based on not only the stakeholder’s views, 
but also the results of needs assessment and social science theories. However, conceptualization is rarely the 
focus of evaluation of established programs since their conceptual frameworks already exist and are less likely to 
change. (Shadish, 1991, p. 404) Instead, an evaluator will take a more summative approach and much of his/her 
responsibility falls into judging the program accountability. The role of an evaluator is less summative in 
fine-tuning programs, with emphasis on identifying the needs for change and formative modifications. 

Rossi’s attempt to integrate the works of various theorists into one theoretical framework also helps to shape his 
position on the issue of the proper role for an evaluator. Rossi appreciates the strengths of different roles 
proposed by other theorists and assign such roles to the contexts that best fit them. The “good enough rule” of 
doing evaluation proposed by Rossi also frees evaluators from the everlasting debate of internal validity vs. 
external validity, quantitative methods vs. qualitative methods, descriptive value vs. prescriptive value. Instead, 
it allows evaluators to choose the best possible design through assessing all kinds of “trade-offs”. 

Strength and Weakness of Rossi’s position: In my opinion, Rossi’s stance on the role of evaluator is closest to 
the reality, since evaluation by nature is highly context-based. The nature of different social programs, clients’ 
expectations, evaluators’ backgrounds and expertise, the employment status, available resources, and restraints, 
as well as the influence of culture and politics can all result in quite different approaches towards doing an 
evaluation. Even for the same evaluation, activities in different stages will require various competencies from an 
evaluator. As a result, there shouldn’t be only one proper role for an evaluator and Rossi’s attempt to integrate 
different roles into one theoretical framework is plausible. However, Rossi is less explicit when linking a 
specific context to a specific role and linking the specific role to a set of responsibilities that relates to such role. 
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2.6 A Comparative Analysis of Different Theorists ’ Positions 

Looking beyond the different metaphors used by theorists to express their opinions on the roles of evaluator, a 
comparative analysis will be done in this section to dissect such roles into specific behaviors that an evaluator 
should do in different phases of an evaluation. Those phases are listed in the table below as: program selection, 
criteria selection, data collecting, evaluation findings, evaluation use and dissemination. (See table 1) As we can 
see from the table, different roles an evaluator takes can result in quite different approaches of doing evaluation 
in certain phases while still share a lot of similarity in the other phases. All the citations in the following table are 
from the book Foundations of Program Evaluation. (Shadish et al, 1991) 

Insert Table 1 Here 

3. Proposed Resolution of the Fundamental Issue 

The current debate among evaluation theorists regarding the proper roles for an evaluator reflects their different 
stances on other fundamental issues such as the value of evaluation (descriptive vs. prescriptive), the methods of 
evaluation (quantitative vs. qualitative), the use of evaluation (instrumental vs. enlightenment), the purpose of 
evaluation (summative vs. formative). My own resolution on the issue of evaluator’s role will also be based on 
my understanding of those fundamental issues. 

Value: an evaluator should prioritize the values from different stakeholder groups when selecting the criteria of 
merit for evaluands. Not only does the descriptive values approach reflect the concept of a plural democracy, it 
also orients the evaluation questions towards the concerns of its stakeholders thus makes it more likely that the 
evaluation finings will be used by them. By taking into account different opinions regarding the values of the 
program, an evaluator can also reduce the personal bias, which is hard to avoid when the evaluator is the one 
who assigns values. However, in case different stakeholders can’t reach an agreement on the issue of value, an 
evaluator should take measures to make his/her judgment about the program value. For example, the evaluator 
can conduct a needs-assessment to identify the primary stakeholder group who will be affected most by the 
program and prioritize their values when selecting the criteria of merit for the evaluation. 

Methods: an evaluator should be familiar with quantitative and qualitative methods and accept them both as 
available methods for conducting evaluation. However, an evaluator should meet with clients before evaluation 
and get their opinions on the suggested method. If the clients have a strong opinion to get some quantitative data 
and have some nice charts in the final report, then the case study is not an ideal method. If the clients want the 
program to suffer from minimum intrusion from the evaluation, then experiment or quasi-experiment design 
shouldn’t be considered as first options. However, this is not to say that evaluators should discard the methods 
they consider best for the evaluation immediately if such methods are not accepted by clients; but it is essential 
for evaluators to convince their clients of the proposed methods for evaluation and reach an agreement before 
applying any method. 

Use: an evaluator should emphasize the instrumental use of his/her evaluation findings and actively promote the 
dissemination of the evaluation results. It is hard to imagine that an evaluation will be initiated with no intent to 
know the effect of a social program, especially when such program costs a lot of tax-payers’ money and impacts 
a large population. In my opinion, ignoring the instrumental use of evaluation is highly irresponsible and has 
detrimental impact on the profession of evaluation: why should I hire someone to evaluate a program if he/she 
cannot tell me if the program is working or not? The enlightenment use of evaluation sounds promising but has 
its limitation in actual practice. First, it is hard to determine the scope of data collecting. With potential users of 
evaluation findings in obscurity, it is hard to know what specific data will be useful for them. Second, it is hard 
to enlighten people without telling them what works in the program and what does not. We have to make a 
judgment about success and failure if we want others to learn from the success or the failure. Last but not least, 
clients’ expectation and time restraint often make conducting evaluation for enlightenment impossible. It takes 
time to measure the program input, implementation, outcome, long-term impacts peoples’ attitude, etc. But most 
clients do not have so much time for an evaluation. 

Purpose: I prefer a more summative role for an evaluator for the following reasons. First of all, an evaluator is 
hired for his/her expertise in making rational judgments. If clients want only the vicarious experiences or a set of 
quantitative data, they can just hire some writers or statisticians to do the job. Second, even for the purpose to 
make a program better there is a difference between an evaluator and a program manager. At some point, an 
evaluator will give a summative opinion about what works and what does not in the program. An evaluator can 
make recommendation about how to make improvement; but ultimately it is up to the program manager to make 
that decision based on his/her knowledge about the program such as the available resources or people’s 
commitment. Lastly, being too involved in improving a program might hinder the objective and rational 
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judgment from evaluators. A lot of personal bias is expected if the evaluator of a program becomes its advocate; 
since it is hard for an evaluator to make a statement that a program is not working and should be terminated if 
the evaluator believes in such program and has invested heavily in it. 

However, by reviewing other theorists’ position on the issue of evaluator’s roles, we can see each different role 
will have its unique strength and weakness and there is no specific role that fits all the evaluation. Instead of 
using another metaphor to describe my position, I will define the role of evaluator in the things an evaluator will 
do during an evaluation, as shown in the following table. (See table 2) 

Insert Table 2 Here 

Nevertheless, 1 understand my resolution for such fundamental issue is not perfect and is vulnerable to criticism 
in several aspects. First of all, the role I proposed for evaluators might not work in certain contexts; or it is very 
likely that evaluators who take different roles can still achieve great successes in conducting evaluations. Take 
Stake, Weiss, Wholey and Cronbach for example, their difference regarding the proper role for an evaluator 
doesn’t prevent them from presenting strong evaluation cases to support their positions. In other words, it is 
almost like I propose a role for an evaluator and then caution evaluators to take such role while in the same time 
indicate other roles might work even better in their contexts. Second, different cultural, racial and ethnic 
backgrounds of evaluators could also affect their judgments about issues such as clients’ expectations, dynamics 
between the program administrators and evaluators, the monetary relationship between evaluators, program 
administrators, federal government and stakeholder groups, as those issues might vary significantly in different 
background contexts. Since my resolution is based on my judgment of such issues, if my judgment is an 
incorrect reflection of the reality elsewhere, then my resolution might not fit well in that context. Last but not 
least, people can also attack the logic behind my approach to define the role for an evaluator. Maybe it is wrong 
to define the evaluator’s role in terms of what they should do during the evaluation; maybe an evaluator’s role 
should be associated with the purpose of an evaluation and such role actually defines what an evaluator should 
do in the evaluation. 
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Notes 

Note 1. Readers gain new perspectives and ideas from reading the evaluation reports, which results in 
accumulated knowledge and evidence that can be used by them in many ways including defining a problem, 
suggesting a solution and making a policy. Differing from the instrumental use of evaluation results, 
enlightenment emphasizes the long term effect on policy-making. 
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Table 1. A summary of theorists' positions on the issue of evaluator's role 



Scriven 

Campbell 

Stake 

Weiss 

Rossi 

Program 

selection 

“everything” 

(p. 84) 

Preference for 
pilot 

programs 

(P-136) 

Preference for 
local programs 

Programs whose 
evaluation 
results are likely 
to be used 

Divide programs into 3 
kinds: innovative, 
established and 
fine-tuning 

Criteria 

selection 

Prescribe value 
through needs- 

assessment 

Describe 

values from 
political 

process 

(P-160) 

Describe values 

from local 

stakeholders 

(p.307) 

Describe values 

from different 

stakeholders 

(P-210) 

Prefer criteria to come 

from stakeholder 
agreement but unclear 
how to avoid biases 

Data collecting 
scope 

The outcome effects 
of the program 

Outcome 

consequence 

of the 

program 

(p. 161) 

13 kinds of data 
covering the 
program 

antecedents, 
transaction and 

the outcome 

(p.282) 

Not only the 
immediate 

outcome of the 
program, but 
also its input, 
implementation 
and long-tenn 
outcome (p.205) 

Collecting data 
regarding the program 
implementation, 
outcome as well as its 
efficiency 

Data collecting 
method 

Not specific 

Randomized- 
experiments 
or strong 
quasi-experim 

ents 

Qualitative 
methods, case 

studies 

Both 

quantitative and 
qualitative 
methods 
(p.204-205) 

Accept both 
quantitative and 
qualitative methods 
with a preference 
towards the fonner. 

Evaluation 

findings 

One final evaluative 
judgment regarding 
the program effects 
and “need” 

Reports about 
the causal 

inference of 

program 

effect 

Accurate 
portrayal of the 
program, diverse 
formats 
(p.282-283) 

Separate 
summaries for 

different 

stakeholder 
groups, with 
knowledge and 
information that 

best interest 

them. 

Include not only the 
program effects, but 
also its policy 
relevancy such as 
cost-efficiency, 
alternative options, etc. 
(p.409) 

Use of 

evaluation 

results 

Instrumental use of 

the results to choose 

the better evaluand 
or improve one. 

(p. 109) 

Instrumental 

use of the 

results to 

solve a social 
problem. 

Provide 

vicarious 
experiences of 
the program 

Enlightenment 
use of the results 

to accumulate 
knowledge and 
shape 

policy-making 

Instrumental, 
conceptual and 
persuasive use of the 
results, (p.410-411) 

Dissemination 

of the 

evaluation 

results 

Not mentioned 

Dissemination 

is not the 

concern of an 

evaluator 
(p. 162) 

Not mentioned 

Actively 
facilitate the 

dissemination of 
results (p.207) 

“ a definite 
responsibility of 
evaluation researchers” 
(P-410) 
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Table 2. A summary of my position on the issue of evaluator's role 


Phase of Evaluation 

Evaluator’s responsibility 

Program selection 

• Determine the nature of the program 

• Analyze the political influences that affect the program 

Criteria selection 

• Use values from different stakeholder groups 

• Conduct needs-assessment to prioritize the values of different stakeholder groups 

Data collecting scope 

• Collect data that can answer stakeholders' inquiries and concerns 

• Emphasis on the program outcome and look into program input and implementation 

Data collecting methods 

• Get to know clients' expectation and available resources 

• Be familiar with both quantitative and qualitative methods 

Evaluation findings 

• Provide answers to the inquiry of different stakeholder groups 

• Offer evaluative judgment regarding the program effect 

Use of evaluation results 

• Use the results as evidence to make decision regarding the program (promote, modify 
or terminate the program) 

• Increase people’s knowledge of certain phenomenon 

Dissemination of evaluation 

results 

• Provide separate summaries of evaluation results to different stakeholder groups with 

information about their interest and concern 
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